Skip to content

Qwen3.5-4b BF16 optimization and correctness (sm_120) - #266

Merged
localai-bot merged 17 commits into
mudler:mainfrom
richiejp:row/KERNEL-SSM-MAMBA
Aug 11, 2026
Merged

Qwen3.5-4b BF16 optimization and correctness (sm_120)#266
localai-bot merged 17 commits into
mudler:mainfrom
richiejp:row/KERNEL-SSM-MAMBA

Conversation

@richiejp

Copy link
Copy Markdown
Contributor

Refs #206
Closes #237

Row

KERNEL-SSM-MAMBA — Qwen3.5-4B BF16 optimization and correctness work for consumer Blackwell (sm_120).

Before starting

What changed

Improves the Qwen3.5-4B BF16 path on an RTX 5070 Ti (sm_120) while preserving exact output. The branch adds pretokenized benchmark input and atomic request-wave
admission, retains the accepted GDN optimizations, records and removes rejected experiments, and updates the benchmark evidence.

It also fixes #237: the CUDA BF16 MoE resident cache was keyed by a reusable MoeBlockWeights wrapper address and could return stale device pointers after engine
teardown. The cache is now keyed by the uploaded router allocation and retains its allocation owner. A deterministic repeated-engine regression covers both identity
reuse and owner lifetime.

Evidence

  • scripts/agent-preflight.sh passes

    • Full preflight still reports established unrelated baseline failures, including test-registration/live-row checks and sandboxed release-fixture failures.
    • Commit trailers, document checkpoint, NOW checkpoint, diff check, and PR-size checks pass.
  • Focused CUDA gate passes at 1d8cfcbed33868bda50f6be66b2f3655cbae545e:

    nix develop .#cuda --command flock /tmp/gpu -c \
      "ctest --test-dir build-nix-cuda --output-on-failure \
      -R '^(test_engine_core_proc|test_async_llm|test_bench|test_gdn_decode_fused|test_gdn_prefill_conv|test_ops_gdn)$'"
    

Result: 6/6 passed.

  • test_bench repeated-process stress: 50/50 passed.

  • Independent mutation review:

    • Restoring the old MoeBlockWeights* cache key fails with invalid identities and zero-token output.
    • Removing the retained allocation owner fails with 20 invalid cache entries.
    • Restored implementation passes; reviewer verdict: PASS with no findings.
  • Same-change records are updated in docs/STATUS.md, docs/BENCHMARKS.md, docs/FEATURES.md, .agents/NOW.md, and the owning specs.

Speed claims

  • The operator ran the measurements under the GPU lock and recorded them in docs/BENCHMARKS.md.

Corrected Qwen3.5-4B BF16 comparison against production vLLM:

  • Total/output throughput: 1.0283x vLLM — 2.83% ahead.
  • TTFT: 1.0853x slower.
  • TPOT: 1.0165x slower.
  • E2E latency: 1.0288x slower.
  • VRAM: 118.7 MiB higher.

Honest gaps

  • TTFT, TPOT, E2E latency, and VRAM parity remain open under RTX 5070 Ti: close Qwen3.5-4B TTFT, TPOT, and VRAM gaps vs vLLM #206.
  • The retained GDN optimizations improve the measured timing axes but do not close every gap.
  • Rejected argmax-scratch, vector-writeback, and pre-serve-warmup experiments are documented and removed from product code.
  • Full repository preflight remains red on pre-existing unrelated checks; no waiver was added.
  • The process-lifetime BF16 MoE cache retains one small router allocation for each cached engine identity.

@richiejp
richiejp force-pushed the row/KERNEL-SSM-MAMBA branch 2 times, most recently from f1f952c to 7ee42f6 Compare August 10, 2026 14:11
Import the measured optimization history and bind issue mudler#206 to a no-regression campaign for throughput, latency, and memory.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Record the accepted combined local A/B, void mismatched cross-engine timing ratios, and commit the pretokenized-admission contract before implementation.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Match the pinned vLLM benchmark frontend by encoding all prompts before the timed interval and admitting token IDs by default. Keep VT_BENCH_PRETOKENIZE=0 as the exact timed-string rollback and report the resolved path.

Focused CPU gate: test_bench 6/6 tests, 57/57 assertions.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Make benchmark preparation own special-token encoding before clock start, and strengthen the synthetic fixture so raw encoding cannot masquerade as InputProcessor parity. Capture both resolved report modes.

Focused gate: test_bench 10/10 tests, 70/70 assertions.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Record the failed real pretokenized A/B as correctness-void and bind the next repair to atomic AsyncLLM/core-queue wave publication.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Prepare and register complete AsyncLLM string/token waves before one all-or-zero core queue publish, and make the benchmark use that boundary for initial and refill admissions.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Keep argmax scratch per CUDA device and stream, grow it geometrically with stream-ordered allocation, and retire graph-visible blocks. Preserve the exact incumbent allocator unless the strict experimental selector is enabled. Add portable ownership, failure, capture, isolation, and serialization tests.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Route cuda_sample through the capability-limited ownership seam and harden transactional first allocation plus reviewer mutation coverage.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Remove the geometric argmax scratch and BF16 GDN vector writeback candidates after same-binary profiles miss their Pareto gates. Record the corrected atomic-wave baseline, retained local opt-ins, and falsified trace evidence.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Ground the remaining Qwen3.5 TTFT deficit in the first request wave and specify a strict opt-in pre-serve BF16 warmup discriminator mirroring vLLM's mandatory profile run.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
Record the exact cold-versus-warm TTFT discriminator and remove the opt-in after it misses both prespecified magnitude gates.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-5.6-sol [Codex]
@richiejp
richiejp force-pushed the row/KERNEL-SSM-MAMBA branch from 7ee42f6 to a0a8a5b Compare August 10, 2026 16:00
@localai-bot
localai-bot merged commit ae78871 into mudler:main Aug 11, 2026
12 of 13 checks passed
localai-bot pushed a commit that referenced this pull request Aug 11, 2026
main advanced by ten commits while the six external PRs were being reviewed and
merged (SILU-VECTORIZE, SERVE-METRICS-ASYNC #277, SAMPLE-LOGPROBS-RECORD-REPAIR
#259, PERF-27B-LMHEAD-DSR #298, PERF-35B-SHARED-DOWN-BF16, ENG-OPERATOR-RECORD-
COVERAGE-FIX #304, release #303). Resolved rather than re-based so nothing of
either side is lost.

  - .agents/roadmap_v1.md: issue rows are DISTINCT keys, so the two sides union
    -- main's #298 alongside this landing's #269/#305/#206.
  - .agents/NOW.md: main's 35B mid-band row is NEWER (two levers landed, not
    one) and was taken wholesale; only the Qwen3.5-4B row was reapplied on top,
    with main's row verified byte-present afterwards. The "next actions" hunk
    kept this side's item 3, which post-dates main's copy.
  - .agents/benchmark-record.md union-appended, main's entries first.
  - tests/vllm/v1/test_async_llm.cpp: both sides add a distinct `using`
    declaration (PublishAsyncRequestWaveIfAlive from #266, PrometheusStatLogger
    from #277) -- union, not a choice.
  - scripts/check-public-doc-tables.py: main had re-pinned the STATUS ratchet to
    243512 for #277 while this landing had reconciled #155 and #266 to 243455.
    Both rationale histories are kept and the value re-MEASURED on the merged
    page: 243368. Strictly down, the only direction it may move; the byte-tight
    test asserts equality, so this is measured, never predicted.

Doc-table checker and its 54 unit tests green; clean CPU build.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]
localai-bot pushed a commit that referenced this pull request Aug 11, 2026
…rompt

Landing #266 alongside main's #277 turned test_async_llm RED:

  test_async_llm.cpp:915
  CHECK( MetricValue(t, "vllm:prompt_tokens_total"...) == 1.0 )
  values: CHECK( 2 == 1 )

Root cause is a SHARED TEST FIXTURE, not product code. #266 changed
BuildFixture() -- used by all 15 cases in the file -- to register a <tool>
special token and a TemplateProcessing post-processor that prefixes it to
EVERY encoded prompt. That silently makes every prompt one token longer, so
#277's "depth-2 batch-queue step folds IterationStats" case, which asserts
the exact count vllm:prompt_tokens_total == 1 for the prompt "hello", sees 2.

Neither side's CI could see it: #277's test did not exist when #266 was
authored, and #266 was not merged when #277 landed. It only appears once both
are in one tree.

The fix keeps both behaviors instead of choosing between them:
BuildFixture(bool with_special_post_processor = false). The default now
reproduces main's fixture byte for byte, so the 14 pre-existing call sites are
inert; the one case that genuinely needs it -- "ordered waves preserve
string/token inputs and one-item behavior", which asserts
EncodeWithSpecialTokens != Encode -- opts in explicitly. A comment on the
signature records why it must stay opt-in, so the next person to reach for a
global fixture change sees the failure mode first.

Verified: test_async_llm 15/15 cases, 440/440 assertions, clean build. A first
attempt that simply reverted the fixture wholesale was REJECTED because it
broke that wave case at :350 -- the post-processor is genuinely needed there,
just not everywhere.

Two measurement traps hit while root-causing this, recorded so the evidence is
not mistaken for more than it is. An early bisect reverted the engine but not
the test file; the rebuild failed with 39 errors and the STALE binary appeared
to reproduce the failure, which would have blamed the wrong files. And a later
rebuild failed with "No space left on device" (the box root filesystem is at
100%, 425G/447G), which reads exactly like a compile error. Both were caught
only by checking the build exit code before trusting the test output.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]
localai-bot pushed a commit that referenced this pull request Aug 11, 2026
…nel-tile spike

.agents/specs/sm120-qwen35-conv-channel-tile-2026-08-08.md carries four raw
conflict markers on main (lines 4/7 and 144/160, tagged `pr-266`), left behind
when #266 and #155 landed. Both hunks have an EMPTY incoming side, so the
resolution is to keep the HEAD text and drop the markers: the issue/PR
attribution line, and the paragraph recording that the arm-2 preload NARROWS
the stripe-overwrite window rather than closing it, with the residual hazard
tracked as #305.

That second paragraph is why this is a commit and not a silent tidy: it is the
spike's honest-limits statement, and a reader who stops at the `<<<<<<<` line
never reaches it.

No content is added or removed beyond the four marker lines.

Refs #266, #155, #305.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CUDA BF16 MoE resident cache reuses stale pointers after engine teardown

2 participants